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ABSTEACT 

The paper concerns instruments used in the 
determination of task performance efficiency of mechanical and 
electronic maintenance men. Composite results of unvalidated 
knowledge tests, theory +ests^ school marks, and supervisors' ratings 
are presented which indicate that these measurement instruments have 
low empirical validity. The limitations of traditional systems 
effectiveness measures are also discussed. Although job task 
performance tests (JTPT) have higher empirical validity, they have 
had limited use due to their relatively high cost factors. A series 
of efforts to develop better criterion referenced JTPT^ and to 
attempt the development of paper and pencil symbolic substitute tests 
of high empirical validity^ were made. Described is the model battery 
of 48 Criterion referenced JTPT covering all key maintenance 
activities (checkout^ align/adjust, remove/replace, troubleshooting ^ 
test equipment, and soldering), intended for use on the job^ in 
trainingr and for validation of symbolic substitute tests. Developed 
batteries of graphic and video symbolic substitute tests were given 
limited validations. Suggestions are made for use of the criterion 
referenced JTPT in improving maintenance efficiency. Fourteen 
references are listed, and an appended section describes limitations 
of system effectiveness measures. (Author/MS) 
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CRITERION REFERENCED MEASURES OP TECHNICAL PROFICIENCY 
IN MAINTENANCE ACTIVITIES 



John P. Foley, Jr. Ed.D. 



Air Force Human Resources Laboratory 
Wright-Patterson Air Force Base, Ohio 



ABSTRACT 

The piper concirns Instrumtnts used in tht ditirmi nation of how tffeclently 
mainttm^nce men ptrform the various tasks of their jobs. Currently, a grtat deal of 
reliance is placed on unvalidated job-knowledge tests, theory tests, school marks, 
supervisors' ratings for such determination. This paptr prtsents a composite of the 
results of many research and development tfforts concerning tht empirical validity of 
such instruments. These results indicate that thest measurement instruments have 
low empirical validity. The limitations of traditional systems effectivtness measures 
art also discussed. Although job task performance tests (JTPT) have much higher 
en^irical validity, such tests have had very limited use because of their relatively high 
costs in time, personnel and equipmtnt. As a result of the foregoing findings, the Air 
Force Human Resources Laboratory (AFHRL) supported a series of efforts to 
develop bener criterion referenced JTPT. and to attempt the development of paper 
and pencil symbolic substitute tests of high empirical validity. The paper describes 
the model battery of 48 criterion referenced JTPT, which has been developed to 
cover all key mainttnance activities such 9i checkout, align/adjust, remove /replace, 
troubleshooting, test equipment and soldering. During this development many factors 
were considered including the identification and classification of tasks to be 
measured, the hierarchal rtlationship of maintenance tasks, the most effective order 
of their meesurement and the ease cf test administration. This battery was developed 
as a mot$ci of JTPT to be used on the Job and in training. It was also intended as a 
battery of criterion tests for the validation of paper and pencil symbolic substitute 
tests. Batterfes of graphic and video symbolic substitute tests wert developed and 
given limited validations. The validation of graphic symbolic substitute tests indicated 
that the symbolics for all activities, with the exception of soldering, have promise. 
The paper discusses the requirements for additional refinement and validation for the 
various graphic tests. An unsuccessful effort to develop video symbolic substitute 
tests Is eiso mentioned. Suggestions are made for tfie application of criterion refer- 
enced JTPT for the improvement of maintenance efficitncy. 



INTRODUCTION 

Human factors specialists can point with great pride to 
their achievements In the area of the interface between ma- 
chines and humans. However, these achievements involve 
primarily the operators of the hardware system and not the 
maintenance of the system. 

Part of this emphasis on the human operator seems to be 
due to the fact that the operator has much greater visibility. He 
performs when the machine performs It is quite obvious that 
the performance of the system usually is directly related to the 
operator's performance. 

Many actions are taken to maximize effective and efficient 
performance of the operator. Work stations are human engi- 
neered to maximize the efficiency and comfort of the human 
operator. Major training facilities are provided so that operators 
can enjoy a large amount of supervised practice in performing 
typical tasks of his job. Graduation from training is based 
primarily on demonstrated ability to perform job tasks. And, 
periodic checks are made of the operator's abiiity to perform 
the critical tasks of his job. These, of course, are not all of the 
many efforts made to maximize the performance of human 
operators. 

The important contributions of the maintenance man to 
the successful performance of the man-machine system are less 
visible than the operator's contribution. Most of the mainte- 
nance man's performance takes place while the man-machine 
system is not performing its mission. The effective end-product 
of the maintenance man's efforts, which are performed during 



this downtime, is an effectively operating machine for Its next 
mission. There is no doubt that such maintenance must be 
effective. But, given sufficient resources, maintenance need not 
be efficient to be effective. And, at present, very little is done 
to ensure such efficiency. 

SYSTEMS EFFECTIVENESS MEASURES 

A critical examination of systems effectiveness parallels 
the above discussion. For the man-machine system to be effi- 
cient and reliable during a mission, the machine subsystem must 
be efficient and reliable during the mission. The maintenance 
man makes his contribution to these measures before the 
performance of each mission by ensuring that the machine sub- 
system is in prime operating condition when the mission is 
started. This contribution must be effective or the mission 
probably will not begin. 

Once the mission starts, the efficiency of the system 
depends on the performance of both the machine and its 
operator(s). If the mission succeeds in an efficient manner both 
have operated efficiently. However, if the mission is performed 
inefficiently, the cause may be either the human subsystem or 
the machine subsystem. And, if the inefficiency Is traced to the 
machine subsystem, it may be further traced to ineffective 
maintenance. The faulty maintenance can be corrected. 
However, the major point is that even when faulty maintenance 
is discovered and corrected, it need only be effective and not 
necessarily efficient. The usual measures of system effectiveness 
include indices of the effectiveness and efficiency of the human 
operator, but they represent only the effectiveness, and not the 
efficiency, of maintenance personnel. (A more detailed dis- 
cussion of the relationships between systems effectiveness 
measures and the efficiency of maintenance is presented in 
Appendix A.) 

This acceptance of effectiveness, without efficiency seems 
to "set the stage" for the acceptance of Inefficiency in mainte- 
nance. Unfortunately, methods used to select, train and 
promote maintenance personnel also contribute to inefficient 
maintenance. 

MAINTENANCE PERSONNEL 
EFFECTIVENESS MEASURES 

The personnel system, which includes formal training, has 
developed its own measures for making initial selection, for 
ascertaining effectiveness of training and for the promotion of 
maintenance personnel. Most of these measures Involve tests 
which are of the paper-and-pencll, job-knowledge variety. The 
effectiveness of formal training ^or x\\^ mechanical maintenance 
specialities is measured mainly by scores obtained from such 
paper-and-pencil job-knowledge tests, even though the students 
in these training programs have received at least some 
"hands-on" practice on many mechanical maintenance tasks. 



The measures of effectiveness of format training programs for 
the vh'i'tronh maintenance specialisties include scores from 
paper-and-pencil job-knowledge tests, as well as theory tests. 
Students in these electronic maintenance courses receive little if 
any "hands on" practice in their maintenance tasks. 

The selection tests for both mechanical and electronic 
maintenance specialities have been standardized against com- 
posite scores from paper-and-pencil tests. This means that the 
people selected for the maintenance specialities have been 
selected not on their aptitude for performing the tasks of their 
maintenance jobs but on their aptitude for making high scores 
on paper-and-pencil, theory and job-knowledge tests. 

The speciality knowledge test (SKT) and the promotion 
fitness examination (PFE) used for advancement up the mainte- 
nance career ladders also are paper-and-pencil job-knowledge 
tests. At the present time, throughout his whole career, a 
maintenance specialist is not required to demonstrate on formal 
job task performance tests (JTPT) that he can perform effi- 
ciently and effectively the tasks of his job (Foley, 1974). 

This state of affairs would not be so bad if there was a high 
empirical relationship between scores received on paper-and- 
pencil theory and job-knowledge tests and scores received on 
JTPT, but this is not the case. Research evidence gives a rather 
low rating to all of these paper-and-pencil tests. Table 1 shows 
correlations that have been obtained by comparing job task 
performance tests (JTPT) to theory tests, and to job-knowledge 
tests. The latter two are paper-and-pencil tests. Table 1 also 
includes correlations of JTPT with school marks. As indicated 



Table 1. Coirelations Between Job-Task Peiformance Tests and Theory 
Tests, Job Knowledge Tests, and School Maiks 



Typ« of Job T»«k 
Ptrforminct Ttit (JT^T) 



Theory 
Tttts 



Job Knowl- 
tdft TnU 



School 
Marks 



Anderson 
(1962a) 

Evans and 
Smith (1953) 

Mack it tt 
at.. (1953) 

Saupt (1955) 

Brown et al., 
(1959) 



Wtiiianns and 
Whitmort 
(1959) 



Test Equipmtnt JTPT 



Troubleshooting JTPT .24 & .36 .12 & .10 



Troubitshooting JTPT .38 

Troubleshooting JTPT ,55 

Troubleshooting JTPT .40 
Ttst Equipment JTPT .29 
Alignment JTPT .28 
Repair Skills JTPT J9 
Troubleshooting JTPT 



.35 
.39 
.56 



Crowdtr tt 
at.. (1954) 



(intxptritnced Subjtcts) 
(Exptritnced Subjtcts) 


.23 
.15 




Adjuitmtnt JTPT 
(inexptritnctd Subjtcts) 
(Exper itnctd Subjtcts) 


.02 
.21 




Acquisition Radar JTPT 
(intxperltrwtd Subjtcts) 
(Experttnctd Subjtcts) 


.03 
.14 


.36 
.22 


Targtt Tracking Radar JTPT 
(Intxptritnctd Sut^tcts) 
(Exptfitnctd Subjtcts) 


.24 

.20 


.33 
.38 


Missilt Tracking Radar JTPT 
(Intxptritnctd Subjtcts) 
(Exptritnctd Subjtcts) 


.09 
.19 


.15 

.32 


Computtr JTPT 
(Inexptritnctd Subjtcts) 
(Experitnctd Subjtcts) 


.06 
.06 


.24 
.14 


Total JTPT 
(Intxptritnctd Subjtcts) 
(Exptrltnctd Subjtcts) 


.14 

.20 




Troubltshooting JTPT 


.11 


.18-.: 



earlier, school marks have been heavily weighted with the 
paper-and-pencil test scores. An examination of this table 
indicates that the correlations of JTPT scores with theory test 
scores are generally somewhat lower than with job-knowledge 
tests. None of these measures are sufficiently valid for use as 
substitutes for JTPT (Foley, 1974). 

A conclusion that can be drawn from the discussion, thus 
far, is that there is nothing in the job structure of individuals in 
the maintenance subsystem that requires them to perform 
maintenance tasks efficiently. Systems effectiveness measures 
requires effective, but not efficient, performance of n^ainte- 
nance tasks. The paper-and-pencil measures used for selection, 
training and promotici require no actual performance of 
maintenance tasks. Wallace (1965) expressed this problem very 
succinctly as it applies to electronic maintenance personnel. A 
somewhat similar statement could be made for mechanical 
maintenance personnel. 1 quote: 

All of this is prelude to my main thesis which is in no 
sense revolutionary, original, or controversial. I state it because 
it is honored in the breach. It is that the nature of our profi- 
ciency measures determines how we select, classify, training, 
maintain and assess our human resources. If the measures are 
largely irrelevant to the jobi we want done, we will select the 
wrong men, classify them incorrectly, and train them wrong. 
This is true because these proficiency measures are, or should 
be, the criteria against which we validate our selection and 
classification procedures and evaluate our training content and 
methodology or our supervisory techniques. Thus, if I use a 
test of advanced electronics theory as the proficiency rneasure 
for electronics maintenance and as the criterion against which 
to evaluate a test for selecting men to go into maintenance 
training, I will end up choosing a selection test which rejects 
men who are not well above average in both reading and 
arithmetic ability. In the process I might reject a great many 
who are outstanding in their ability to' get their hands on a 
piece of machinery and make it work. I might also accept a 
number who (like myself) are so lacking in the simplest 
manipulative ability that their hands could have been cut off 
at the wrists at birth without seriously affecting their outputs. 
So, when I decided what proficiency measures to use, I also 
decided what kind of men I was going to put into training for 
the job. 

But It doesn't end there. For when I now approach the 
problem of how to train men to perform the tasks involved in 
the job, I must make decisions about what should be taught 
and what methods should be used jn teaching it. The only way 
I have of reaching such decisions (except by divination which 
is, admittedly, not a rare procedure) is to measure and com- 
pare the performances achieved with various curricula and 
methodologies. So, in the case of the electronics maintenance 
course, I put in lots of reading about electronics theory and I 
produce graduates who can read and write electronics theory, 
while their equipment deteriorates in hopeless inoperativeness 
(Wallace, 1965, p.4). 

APPLICABLE RESEARCH AND DEVELOPMENT 

Starting in 1969, the Advanced Systems Division of the 
Air Force Human Resources Laboratory supported a modest 
program to provide tbe Air Force with the necessary tools for 
measuring the ability of maintenance personnel to perform the 
key tasks of their jobs. This program had two objectives: (1) to 
develop a model battery of JTPT together with appropriate 
scoring schemes for the measurement of the task performance 
ability of electronic maintenance personnel (an effort was to be 
made for the development of JTI^T which could be easily 
administered), and (2) using this battery as criteria, to develop 
and try out a series of paper-and-pencil symbolic substitute tests 
that would hopefully have high empirical validity. 

Criterion Referenced Job Task Performance Tests 

A model battery of 48 criterion referenced JTPT and a test 
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administrator's handbook were developed for measuring ability 
to perform electronic maintenance tasks. Copies of the actual 
instructions for test subjects together with the test administra- 
tor's handbook are available from the Defense Documentation 
Center (DDC) as AFHRL Technical Report 74-57(11), Part II 
(Shriver et al., 1975). The test administrator's handbook was 
developed with step by-step detailed instructions so that an 
individual with a minimum oS electronic maintenance experi- 
ence can administer the tests. 

The battery includes separate tests for the following classes 
of job activities: (1) equipment checkout (2) alignment/ 
calibration, (3) removal/replacement, (4) soldering, (5) use of 
general and special test equipment, and (6) troubleshooting. 
The Doppler Radar AN/APN-147 and its computer AN/ASN-35 
were selected as a typical electronic system. This system was 
used as the test-bed for this model battery. The soldering and 
general test equipment JTPT are applicable to all electronic 
technicians. The other tests of the battery apply to technicians 
concerned with this specific doppler radar system. A detailed 
description of the development and tryout of these JTPT is 
given in AFHRL-TR-74-57(ll), Part I (Shriver and Foley, 
1974a). 

Each class of activity for which JTPT was developed 
contains its individual mix of behaviors, but it is not mutually 
exclusive. There are dependencies among the classes. As a result 
a four level hierarchy of dependencies can be stated: (1) check- 
out, removal/replacement, and soldering; (2) use of general and 
special test equipment; (3) alignment/calibration; and (4) 
troubleshooting. For example, troubleshooting may include all 
of the activities mentioned before it. 

After considering product, process, and time as to their 
appropriateness for scoring the results for each activity, it was 
decide that a test subject has not reached criterion until he had 
produced a complete, satisfactory product. This was a go, no-go 
criterion. Tests were prepared for seven classes of tasks; namely, 
(1) checkout, (2) physical skill tasks (soldering), (3) renr>ove and 
replace, (4) test equipment, (5) adjustment, (6) alignment, and 
(7) troubleshooting. 

Table 2 summarizes the number of tests, problems and 
scorable products by class developed for the AN/APN-147 and 
AN/ASN-35. The simple addition of numbers shown in Table 2 
indicates that there are 48 tests, 81 problems, and 133 scorable 
products. But, these numbers tell us nothing in terms of the 
content of the tests. To say that one test subject accomplished 
100 scorable products while another accomplished 90, tells us 
nothing about the job readiness of these individuals or that one 
is better than the other. The varieties of scorable products are 
so diverse that any combination of them without regard to what 
they represent is meaningless. The only meaningful presentation 
of such information must be in terms of a profile designed to 
attach meaning to such numbers. A sample of such a profile is 
shown in Figure 1 . 



Table 2, Tests, Problems, and Scorable Products 



ct«» 


Cod* 


Tests 


Problems 


Scorable 
Products 


1. Chackout 


CO 


2 


2 
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2. Physical Skill Talks (soldering) 


PT 


2 


5 


17 


3. Removi and Replace 


RR 


10 


10 


20 


4. Test Equipmint 


SE 


7 


37 


67 


5. Adjustment 


AD 


6 


6 


6 


6. Alignmem 


AL 


10 


10 


10 


7. Troubleshooting 


TS 


11 


11 


11 


Total 
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48 


81 


133 
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figure 1, A profile for displayin| the results obtained by an individua! 
subject from a battery of Job Task Performance Tests concerning an 
Electronic System - the AN/APN-147 and the AN/ASN.35. This repre- 
sents the profile of an individual who has successfully completed most of 
the battery. 

This profile is not presented as the final solution to the 
profile problem for JTPT for electronic maintenance. It does 
contain most of the important Information regarding a test 
subject's success on the full range of tests. It gives a meaningful 
picture of the subject's job abilities as measured by the test 
battery, indicating the subject's strengths and weaknesses. 
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An examination of the profile (Ficjure 1) indicates that 
most of the tests in this battery contain only one problem. For 
example, there are two checkout tests, having one problem each 
and there are eleven troubleshooting tests having one problem 
each. There are two soldering tests; one has two problems and 
the other has three. The voltohmmeter (VOM) test has 20 
problems. 

The subject receives no "credit" for a problem unless he 
obtains all of the expected products. No attempt is made, to 
combine these scores in terms of meaningless numbers. 

The hierarchy of dependencies mentioned previously has 
implication for the order in which tests are administered as well 
as for diagnostics. For example, since troubleshooting includes 
the use of test equipment and other activities in the hierarchy, 
logic would dicate that administration of the tests for the sub- 
activities would precede the troubleshooting tests and that a 
test subject would not be permitted to take the troubleshooting 
tests until he had passed these other subtests. These depen- 
dencies are displayed on the left hand side of the profile. This 
portion of the profile is shown in Figure 2. 

Due to the unavailability of a sufficient number of experi- 
enced test subjects at the time of the tryout of the JTPT 
battery, the tryout was not as extensive as planned. The limited 
tryout did indicate that the tests as developed are adminis- 
tratively feasible. Their continued use, no doubt, would result 
in further modifications and improvements. 

Development of Symbolic Substitutes 

Because of the time, equipment and personnel require- 
ments of some of the JTPT, the availability of empirically valid 
symbolic substitute tests would be highly desirable. Even 
though such tests had not been developed in previous efforts, it 
was our opinion that much more work could be done to 
improve symbolic maintenance tests as substitutes for JTPT. It 
was hypothesized that higher correlations possibly could be 
obtained by a different apfproach to the development of 
symbolic substitute tests. A study of the Tab Tests (Crowder et 
al., 1954, see Table 1) indicated that the JTPT used as the 
criterion measures contained many distractions and inter- 
ruptions to the subject's troubleshooting strategy (cognitive 
process), such as, using test equipment to obtain test point 
information. In addition to such interruptions to the cognitive 
process, the subject can obtain faulty test point Information by 
the improper use of his test equipment. In the symbolic 
substitute Tab Tests, all of these potential pitfalls of the actual 
task were avoided. The subject was given a printed test point 
readout. It was hypothesized that the injection of job equi- 
valent pitfalls into symbolic substitutes possibly would increase 
their empirical validity. 

Based on these hypotheses, a battery of symbolic tests was 
developed under contract with the Matrix Research Company 
of Falls Church VA. A companion graphic symbolic test was 
developed for each of the job activities for which a criterion 
referenced JTPT had previously been developed. Based on two 
limited validations, all of the graphic symbolic tests, with the 
exception of the symbolic test for soldering, indicated sufficient 
promise to justify further consideration and refinement. Table 3 
indicates the correlations obtained from these validations. Due 
to a shortage of available subjects, the number of pairs of 
subject was extremely small. All of these promising graphic 
symbolic tests, therefore, must be given more extensive valida- 
tions using larger numbers of experienced subjects. 

The validation of any such symbolic test requires the 
administration of a companion JTPT as a validation criterion. 
As a result, a validation is an expensive process in terms of 
equipment and experienced manpower. The troubleshooting 
symbolic tests require the nrxjst extensive refinement. Several 
suggestions are made for improving their empirical validity. A 
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TESTS 



COx Chfckout 
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SC5 TV-2 Tube 
Tester 



SEq VOM 

Prob MO 



Prob 11-20 
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TSx Troubleshooting 



Figure 2. The left hand portion of the profile displayed in Figure 1. 
This portion Indicates the hierarchy relationships among the various 
tests of the JTPT battery. 



complete description of these symbolic test efforts can be 
found in AFHRL-TR-74-57(fll) (Shriver and Foley, 1*974b), An 
attempt, also, was made to develop video symbolic substitute 
tests, but this effort produced no promising results. (Shriver et 
al., 1974). 
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Tabid 3. Indicates the Number of Pairs Used as Well 
as the and the Correlations Obtained during Two 
Small Validations of Symbolic Tests 



Ttit ArM 


N 
Pairs 




0 


rt 


Novice Subitctft (Aitus) 










Chock out 


4 


4 00 


1 00 




Reirovt & Replace 


14 


2.57 


^43 




Soldering Tests 


4 


0 


0 




General Test Equip 


6 


2.67 


.67 




Speciai Test Equip 


6 


.67 


.33 




Alignment/Adjustment 


19 


6.37 


.58 




Troubleshooting 


9 


1.00 


-.33 a 




Exp«ri«nc*d Subjects (TAG) 










Overall Troubleshoott 


30 


6.53 


.47 


.68 


Chassis (Black box) 










Isolatbn 


30 


16^3 


.73 


.81 


Stage Isolation 


30 


3.33 


.33 


.46 


Piece/Part Isolation 


15 


.07 


.07 


.16 



' Tills negative correlation was probably due to a miinber of deficiencies such as 
(1) deficiencies in the Fully Proccduralized Job Performance Aids provided tlie sub* 
jecls, (2) deficiencies iit the sequencing of the trouhlesliooiing JTPT in relatinn to the 
sub-tests in the jTPT battery, (31 maintenance difficulties with '.he AN/APN-I47 
AN/A.SN-35 system, and (4) difficulties with the content and administration of test 
i-i|Uipnient pictorials provided in the original troubleshooting symbolic tests. 

Even if graphic symbolic substitutes of high empirical 
validity can be produced, the use of symbolic substitutes will 
never, in my opinion, dispense with the requirement for the 
liberal administration of actual JTPT to maintenance personnel. 
We can never include all aspects of an actual performance of a 
task in a paper-and-pencil symbolic representation of that task, 
but our work indicates that we can come much closer than has 
been done in the past. 

APPLICATION OF JTPT 
IN MAINTENANCE SUBSYSTEMS 

AFHRL-TR-74-57(ll), Part 1, mentioned earlier contains 
many suggestions for the application of criterion referenced 
JTPT in maintenance squadrons and in training (pages 54-56). 1 
will not attempt to discuss alt of these here. Today 1 am going 
to limit my suggestions for the application of this JTPT tech- 
nology for the improvement of the efficiency of maintenance 
subsystems. Any actions taken to improve maintenance effi- 
ciency, should give due consideration to the following points. 

1. At present there is nothing in the structure of each 
maintenance subsystem to ensure that its personnel have the 
ability to correctly perform the maintenance tasks dictated by 
its maintenance concept. Any general weaknesses in such ability 
decrease the efficiency of the maintenance subsystem. 

2. The inability to perform such tasks need not reflect on 
the appearance of maintenance effectiveness provided that un- 
limited materiel resources are available to compensate for such 
personnel weaknesses. 

3. Current systems effectiveness measures consider and 
ensure acceptable effectiveness, but not efficiency, of the 
maintenance subsystem. 

4. The limited amount of available hard data indicates 
that maintenance personnel generally are not able to perform 
many of their critical tasks correctly. 

5. During their careers, maintenance personnel take many 
paper-and-pencil job knowledge tests but few if any JTPT. 
Scores obtained from such job knowledge tests tell us almost 
nothing about how well people can perform tasks of their 
maintenance jobs. (I am convinced that the current unques- 
tioned usage of such paper-and-pencil tests in field and training 
situations would be unforgivable if the people involved really 
understood how invalid their current testing practices really are. 
No matter how cheaply paper-and-pencil job-knowledge. tests 
can be prepared or how easily they can be administered, such 
tests are* not a bargain. Their results are almost meaningless in 



terms of ability to perform maintenance tasks AFHRL-TR- 
74-57(1) contains a detailed discussion of th^ir criteria and 
validity). 

Considering the above fiye points, effective action should 
be taken to modify the structure of each maintenance sub- 
system so as to reward not only effective perfomnance but also 
efficient performance. Maintenance personnel should be 
provided supervised practice in performing key tasks of their 
jobs. 1 am suggesting that a battery of JTPT should be devel- 
oped for each new maintenance subsystem, and that each 
member of the maintenance subsystem should be tested period- 
ically on his ability to perform key tasks of his job. Special 
attention should be given to his ability to use test equipment 
accurately. Sampling techniques can be applied for individuals 
and groups to reduce the time required for such tests. 

I am sure that the first efforts for applying the«s measures 
will meet with the traditional objections - especially that such 
procedures would cost too much in time and money. My answer 
to such objections is that we can no longer afford the luxury of 
unlimited materiel support required by our current maintenance 
subsystem operations. We can no longer afford the luxury of 
personnel in our maintenance subsystems who can obtain high 
scores on paper-and-pencil job-knowledge tests but who are 
unable to perform the critical tasks of their maintenance jobs. 
So- we can't afford not to include JTPT in the structure of 
our maintenance subsystems. The model battery which I 
described briefly today, together with the discussions found in 
AFHRL-TR-74-57(l) (Foley, 1974) and AFHRL-TR-74-57(ll) 
Part I (Shriver and Foley, 1974a) provide the bases for such 
action. 
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APPENDIX A 

LIMITATIONS OF SYSTEM 
EFFECTIVENESS MEASURES 

A critical examination of systems effectiveness measures 
supports the contention that such measures demand mainte- 
nance effectiveness but not necessarily maintenance efficiency. 
Typical systems effectiveness measures are indicated in column 
1 of Table A-1. Column 2 indicates those measures which 
require operator effectiveness. Column 3 indicates those 
measures which require maintenance effectiveness. 
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Table A-1. Indicating the Levds of Performance 
Demanded of the Opentor and Maintenance Portions 
of a Human Subsystem by Traditional Systems 
Effectiveness Measures 



Syitem Effcctivencii 
Mcaiur«» 


OPtratore 


Maintvninc* 
PtrtonntI 


Efficiency of System 

\r\ Perform WI<;«5lnn 

(Accuracy, Time, Etc.) 


E, + E, 




Reliability of System 


E,+E2 




Turnaround Time 






Down Time 




Ea 


Spare Parts Consumption 






Acquisition Cost 






Life Cycle Cost 


C 


C 


E, = Efficiency E, 


- Effectiveness 


C =5 Contribute to 



The table indicates that turn around time, downtime, and 
spare parts consumption are systems measures that reflect on 
maintenance effectiveness. But all of these measures are based 
on a predetermined level of what is considered to be normal 
based on past history. 

Available evidence indicates that such measures are 
probably masking some expensive defects and weaknesses in the 
performance of the human portion of the maintenance sub- 
system. High estimates for these measures, which have been 
made during a system's design, probably cover up these 
weaknesses. For example, levels of consumption of spare parts, 
maintenance actions, down times, and numbers of maintenance 
technicians which are truly excessive are frequently accepted as 
"normal." Such excesses would only become apparent when the 
predetermined levels were not sufficient to cover them. As a 
result, the inefficiencies of maintenance personnel may not 
become immediately apparent during system evaluation, if ever. 
The overall weaknesses of the human maintenance subsystems 
may not be attributed to the maintenance technicians but to 
something else. (The evidence to support these statements were 
presented by the author in a previous paper, Foley 1969. Avail- 
able data indicate that the craftsmenship of a large number of 
maintenance men is poor, that they cannot use test equipment 
effectively, and that they cannot troubleshoot efficiently). Any 
such inefficiency on the part of maintenance men will of course 
increase the life cycle of the man-machine system. 

The above analysis of systems effectiveness measures 
indicates that a man-machine system demands efficient, as well 
as effective performance of the operator portion of its human 
subsystem, but only demands effective performance of the 
maintenance portion of its human subsystem. Available data 
indicates that the actual performance of the maintenar>ce sub- 
system generally fits this pattern. In my opinion, this 
acceptance of effectiveness (without efficiency) is an extremely 
important causative factor, but not the only factor contributing 
to maintenance subsystem inefficiency. 
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Tabic S, Indicates (he Number of Pain Used Well 
as the and the Correlations Obtained durini Two 
Small Validations of Symbolic Tests 



Ttit AIM 


ft 
Fsirt 








Nautc* liihtrti (Aiiim) 














4 00 


1 Cf\ 

1 ,uu 




RMT>OVt ll Rlpl«Ct 


14 


2.57 


.43 




Soldering Tttu 


4 


0 


0 




Gtnirat Tttt Equip 


6 


2^7 


.67 




SpKUl T«t Equip 


6 


.67 


.33 




A)i9nm«nt/Adi u itmcnt 


19 


6.37 


.58 




Troubltthooting 


9 


1.00 


-.33* 




Expwknc*d Subitclt (TAG) 










Ovefill Troubltihoollng 


30 


0.53 


.47 


.68 


ChMm (BiKk box) 










isolation 


30 


16^3 


.73 


.81 


SiA9# ItoUtion 


30 


3^3 


.33 


.46 


P<«ct/P»rt ItoUllon 


15 


.07 


.07 


.16 



Hut ii(((it!vc correction w»i probably <Juc to j tuiinbcr o» Jcricicucic* ^tich m 
pj (icfidcncics in tlx TuUy rroccduraltzcJ Job rctfonmncc AtJt proviJcd tlic sub^ 
i«t».(2) dcfidcncici \n the Kqucncuij of lh< troublcUiootitHf JTIT m rcbhon to the 
Jub iCM* in ihcJUTbittcry.p) maintenance difficulties witluhc ANMVN.|47 
ANMSN .)5 s)Slcm.anJ (4) difficukics wiihtbe cmuciu and adminiMraimn of t«t 
vquipiucnt picioriaU provided in the original troublvshoolinf symbolic test*. 

Even if graphic symbolic substitutes of high empirical 
validity can be produced, the use of symbolic substitutes will 
never» in my opinion, dispense with the requirement for the 
liberal administration of actual JTPT to maintenance personnel. 
We can never include all aspects c1 an actual perfornancc of a 
task in a paper-and-pencil symbolic representation of that task, 
but our work indicates that we can come much closer than has 
been done in the past. 

APPLICATION OF JTPT 
IN MAINTENANCE SUBSYSTEI^S 

AFHRL-TR-74-57(ll), Part I, mentioned earlier contains 
many suggestions for the application of criterion referenced 
JTPT in maintenance squadrons and in training (pages 54-56). 1 
will not attempt to discuss all of these here. Today I am going 
to limit my suggestions for the application of this JTPT tech- 
nology for the improvement of the efficiency of maintenance 
subsystems. Any actions taken to improve maintenance effi- 
ciency, should give due consideration to the following points. 

1. At present there is nothing in the structure of each 
maintenance subsystem to ensure that its personnel have the 
ability to correctly perform the maintenance tasks dictated by 
its maintenance concept. Any general weaknesses In such ability 
decrease the vffivkncy of the maintenance subsystem. 

2. The inability to perform such tasks need not reflect on 
the appearance of maintenance effectiveness provided that un« 
limited materiel resources are availablt.to compensate for such 
personnel weaknesses. 

3. Current systems effectiveness measures consider and 
ensure acceptable effectiveness, but not efficiency, of the 
maintenance subsystem. 

4. The limited anx>unt of available hard data indicates 
that maintenance personnel generally are not able to perform 
many of their critical tasks correctly. 

5. During their careers, maintenance personnel take many 
paper-and-poncll job knowledge tests but few if any JTPT. 
Scores obtained from such job knowledge tests tell us almost 
nothing about how well people can perform tasks of their 
maintenance jobs. (I am convinced that the current unques^ 
tioned usage of such paper-and-pencil tests in field and training 
situations would be unforgivable if the people involved really 
understood how invalid their current testing practices really are. 
No matter how cheaply paper-and-pencil job-knowledge. tests 
can be prepared or how easily they can be administered,'such 
tests are^not a bargain. Their results are almost meaningless in 



terms of ability to perform maintenance tasks. AFHRL*Tf 
74-57(1) contains a detailed discussion of their criteria an 
validity). 

Considering the above fiye points, effective action shoul 
be taken to modify the structure of each maintenance sut 
system so as to reward not only effective performance but als 
efficient performance. Maintenance personnel should b 
provided supervised practice in performing key tasks of the 
jobs. I am suggesting that a battery of JTPT should be deve 
oped for each new maintenance subsystem, and that eac 
member of the maintenance subsystem should be tested perioc 
ically on his ability to perforrfi key tasks of his job. Specii 
attention should be given to his ability to use test equipmer 
accurately. Sampling techniques can be applied for Individual 
and groups to reduce the time required for such tests. 

I am sure that the first efforts for applying these measure 
will meet with the traditional objections -especially that suci 
procedures would cost too much in tinr>e and money. My answe 
to such objections is that we can no longer afford the luxury o 
unlimited materiel support required by our current maintenano 
subsystem operations. We can no longer afford the luxury o 
personnel in our maintenance yjbsystems who can obtain hig) 
scores on paper-and-pencil job-knowledge tests but who an 
unable to perform the critical tasks of their maintenance jobs 
So -.we_ can't afford not to Include JTPT in the structure o 
our maintenance subsystems. The model battery which 
described briefly today, together with the discussions found ir 
AFHRUTR.74-57(0 (Foley. 1974) and AFHRL-TR-74.57(I1] 
Part 1 (Shriver and Foley. 1974a) provide the bases for suc^ 
action. 
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APPENDIX A 

LIMITATIONS OF SYSTEM 
EFFECTIVENESS MEASURES 

A critical examination of systems effectiveness measures 
supports the contention that such measures demand mainte* 
nance effectiveness but not necessarily maintenance efficiency. 
Typical systems effectiveness measures are indicated in column 
1 of Table A-l. Column 2 indicates those measures which 
require operator effectiveness. Column 3 indicates those 
measures which require maintenance effectiveness. 



Table A L Indicating the Levels of Pttfonnaiice 
Deminded of the Opentor and Maintenance Portions 
of a Human Subsystem by Traditional Systems 
Effectiveness Meaturet 



SyiUm efftctlvtntsi 

Mm urn 



Malnttnanc* 



Efficiency of System 
\o Peiform Mission 
(Accuracy. Tirne»Clc.l 

Reliability of System 

Turnaround Time 

Down Time 

Sparc Parts Consumption 
Acquisition Cost 
Life Cycle Cost 



Ej 

E2 

E: 
E: 



G, « EfHciency 



■ EffcctivcncM 



C • Contribute to 



The table indicates that turn around time, downtime, an< 
spare parts consumption are systems measures that reflect or 
maintenance effectiveness. But all of these measures are basec 
on a predetermined level of what is considered to be norma 
based on past history. 

Available evidence indicates that such measures an 
probably masking some expensive defects and wealcnesses in thi 
perfomiance of the human portion of the maintenance sub 
system. High estimates for these measures, which have beei 
made during a system's design, probably cover up the& 
weaknesses. For example, levels of consumption of spare parts 
maintenance actions, down times, and numbers of maintenano 
technicians which are truly excessive art frequently accepted a 
"normal." Such excesses would only b6conf>e apparent when th( 
predetermined levels were not sufficient to cover them. As 
result, the inefficiencies of maintenance personnel nf>ay no 
become immediately apparent during system evaluation. If ever 
The overall weaknesses of the human maintenance subsystem 
may not be attributed to the maintenance technicians but t( 
something else. (The evidence to support these statements wen 
presented by the author in a previous paper, Foley 1969. Avail 
able data indicate that the craftsmanship of a large number o 
nwintenance men is poor, that they cannot use test equipnwn 
effectively, and that they cannot troubltshoot efficiently). An> 
such inefficiency on the part of maintenance men will of coursi 
increase the life cycle of the man-machine system. 

The above analysis of systems effectiveness measure 
indicates that a man*machint system demands efficient, as we) 
as effective performance of the operator portion of its humaf 
subsystem, but only demands effective performance of th< 
maintenance portion of its human subsystem. Available dati 
indicates that the actual performance of the maintenance sub 
system generally fits this pattern, in my opinion, thi: 
acceptance of effectiveness (without efficiency) Is an extremely 
important causative factor, but not the only factor contributing 
to nnaintenance subsystem inefficiency. 
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